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(57) Abstract: A cognitive control framework system for 
automatically controlling execution of an application pro- 
gram having a graphical user interface includes a record- 
ing component, an execution scenario script, and a playback 
component. The recording component is adapted to cap- 
ture user input data and images displayed by the graphical 
user interface during a recording phase of execution of the 
application program, and to analyze the captured user in- 
put data and displayed images to generate an execution sce- 
nario (script) during the recording phase. The execution sce- 
nario may be written in a selected high level language (e.g., 
XML). The playback component is adapted to generate sim- 
ulated user input data based on the execution scenario during 
a playback phase of execution of the application program, 
to input the simulated user input data to the application pro- 
gram, to perform image analysis on images displayed by the 
graphical user interface as a result of processing the simu- 
lated user input data during the playback phase and captured 
displayed images from the recording phase; and to automat- 
ically control execution of the application program based at 
least in part on the image analysis. 
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